In image encoding technologies for a plurality of viewpoints, parallax prediction encoding methods of reducing an information amount by predicting a parallax between images when images of a plurality of viewpoints are encoded and decoding methods corresponding to the encoding methods have been proposed (for example, see NPL 1). A vector indicating a parallax between viewpoint images is referred to as a parallax vector. The parallax vector is a 2-dimensional vector having a component value (x component) in the horizontal direction and a component value (y component) in the vertical direction and is calculated for each of the blocks which are regions divided from one image. When multi-viewpoint images are acquired, a camera disposed at each viewpoint is generally used. In the multi-viewpoint encoding methods, each viewpoint image is encoded using such that a plurality of layers as different layers. An encoding method in which each layer further includes a plurality of layers is generally referred to as scalable coding or hierarchy coding. In the scalable coding, high coding efficiency is realized by executing prediction between layers. A layer serving as a criterion without executing the prediction between the layers is referred to as a base layer and the other layers are referred to as enhancement layers. The scalable coding when layers are configured from a viewpoint image is referred to as view scalable coding. At this time, the base layer is referred to as a base view and the enhancement layer is also referred to a non-base view.
As the scalable coding, there are spatial scalable coding (a method of processing a picture with a low resolution as a base layer and processing a picture with a high resolution as an enhancement layer) and SNR scalable coding (a method of processing a picture with low quality as a base layer and processing a picture with high resolution as an enhancement layer) in addition to the view scalable coding. In the scalable coding, for example, a picture of a base layer is used as a reference picture in encoding of a picture of an enhancement layer in some cases.